The present invention relates generally to networking technology. More specifically, the present invention relates to the caching of data objects to accelerate access to, for example, the World Wide Web. Still more specifically, the present invention provides methods and apparatus by which caching systems may be made to coexist with servers which require user authentication for access.
Generally speaking, when a client platform communicates with some remote server, whether via the Internet or an intranet, it crafts a data packet which defines a TCP connection between the two hosts, i.e., the client platform and the destination server. More specifically, the data packet has headers which include the destination IP address, the destination port, the source IP address, the source port, and the protocol type. The destination IP address might be the address of a well known World Wide Web (WWW) search engine such as, for example, Yahoo, in which case, the protocol would be TCP and the destination port would be port 80, a well known port for HTTP and the WWW. The source IP address would, of course, be the IP address for the client platform and the source port would be one of the TCP ports selected by the client. These five pieces of information define the TCP connection.
Given the increase of traffic on the World Wide Web and the growing bandwidth demands of ever more sophisticated multimedia content, there has been constant pressure to find more efficient ways to service data requests than opening direct TCP connections between a requesting client and the primary repository for the desired data. Interestingly, one technique for increasing the efficiency with which data requests are serviced came about as the result of the development of network firewalls in response to security concerns. In the early development of such security measures, proxy servers were employed as firewalls to protect networks and their client machines from corruption by undesirable content and unauthorized access from the outside world. Proxy servers were originally based on Unix machines because that was the prevalent technology at the time. This model was generalized with the advent of SOCKS which was essentially a daemon on a Unix machine. Software on a client platform on the network protected by the firewall was specially configured to communicate with the resident daemon which then made the connection to a destination platform at the client's request. The daemon then passed information back and forth between the client and destination platforms acting as an intermediary or “proxy”.
Not only did this model provide the desired protection for the client's network, it gave the entire network the IP address of the proxy server, therefore simplifying the problem of addressing of data packets to an increasing number of users. Moreover, because of the storage capability of the proxy server, information retrieved from remote servers could be stored rather than simply passed through to the requesting platform. This storage capability was quickly recognized as a means by which access to the World Wide Web could be accelerated. That is, by storing frequently requested data, subsequent requests for the same data could be serviced without having to retrieve the requested data from its original remote source. Currently, most Internet service providers (ISPs) accelerate access to their web sites using proxy servers.
A similar idea led to the development of network caching systems. Network caches are employed near the router of a network to accelerate access to the Internet for the client machines on the network. An example of such a system is described in commonly assigned, copending U.S. patent application Ser. No. 08/946,867 for METHOD AND APPARATUS FOR FACILITATING NETWORK DATA TRANSMISSIONS filed on Oct. 8, 1997, the entire specification of which is incorporated herein by reference for all purposes. Such a cache typically stores the data objects which are most frequently requested by the network users and which do not change too often. Network caches can provide a significant improvement in the time required to download objects to the individual machines, especially where the user group is relatively homogenous with regard to the type of content being requested. The efficiency of a particular caching system is represented by a metric called the “hit ratio” which is a ratio of the number of requests for content satisfied by the cache to the total number of requests for content made by the users of the various client machines on the network. The hit ratio of a caching system is high if its “working set”, i.e., the set of objects stored in the cache, closely resembles the content currently being requested by the user group.
The network cache described in the above-referenced patent application operates transparently to the client network. It accomplishes this in part by “spoofing” the server from which content is requested. That is, if the requested content is in the cache it is sent to the requesting client platform with a header indicating it came from the server having the original content. Even where the requested content is not in the cache, the cache retrieves the original content from the server for which the request was intended, stores it, and then transmits the content from the cache to the requesting client, again indicating that the transmitted data are from the originating server.
As will be understood, some web servers only allow access to real clients. That is, such servers will not transmit requested content in response to a request from a network cache. Only direct requests from the client are honored. Thus, a connection from a cache is rejected and the request is either sent back with an appropriate message in the HTTP header, or the request is simply not answered. Unfortunately, a subsequent request for the same information will go through the same cache with a similar end result. This problem may be solved for a particular cache by configuring the associated router to bypass requests corresponding to certain client/destination pairs as identified by the packet's HTTP header. That is, the system administrator can add access control lists (ACLs) into the router such that data requests which have previously been identified may be passed through the router without being routed through the associated cache.
However, while this may prove somewhat effective in limited circumstances, it destroys the transparency with which the cache is intended to operate. That is, the system administrator needs to monitor rejected requests and manually reconfigure the router, while users on the client network experience, at least temporarily, frustrating limitations on access to desired content until the router ACL is appropriately modified. Moreover, such a solution cannot work in multi-layer networks which do not share administration. As will be appreciated, this is a significant limitation in that this describes most of the world's networking infrastructure.
The problem with the multi-layer or hierarchical network is that there are likely to be more than one cache in between the requesting client and the destination server storing the requested content. Thus, unless each of the upstream caches and/or routers are configured to bypass certain requests, the connection will continue to be rejected until all of the independent reconfigurations occur. This is clearly not an acceptable solution.
It is therefore desirable that a technique is provided by which requests to servers requiring real client access may be made to bypass all upstream network caches in a manner which is transparent to both users and network administrators.